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The findings from this review do not reflect the full body of research evidence on 
Promoting Acceleration of Comprehension and Content Through Text (PACT). 


What is this study about? 

The study examined the effects of the instructional 
practice Promoting Acceleration of Comprehension 
and Content Through Text (PACT), an approach that 
aims to improve social studies content knowledge 
and reading comprehension. This study took place in 
two middle schools in a near-urban district in Texas. 

Study authors conducted random assignment in two 
stages. First, eighth-grade students were randomly 
assigned to 27 social studies classes taught by five 
teachers. Then, each teacher’s classes were randomly 
assigned to either the intervention or comparison 
condition, such that each teacher taught intervention 
and comparison classrooms. A total of 16 classes 
were assigned to the intervention condition and 1 1 
classes to the comparison condition— all of which 
were included in the analysis. The analysis sample 
contained 203 to 218 students in the intervention 
condition and 1 19 to 126 students in the comparison 
condition, depending on the outcome. 

In teachers’ intervention classrooms, they imple- 
mented the five components of PACT during three 
distinct 1 0-day text-based instructional units (Colonial 
America, Road to Revolution, and Revolutionary War). 
In teachers’ comparison classrooms, they provided 
the same social studies content as in their interven- 
tion classrooms but did not explicitly use the 
PACT components (though there were some ele- 
ments of the PACT intervention that were observed 
in the comparison classrooms as part of typical 
teaching practice). 


The study assessed the effectiveness of PACT by 
comparing the general reading comprehension, social 
studies reading comprehension, and social studies 
knowledge of students in the intervention and com- 
parison groups at the end of three instructional units. 


Features of Promoting Acceleration of 
Comprehension and Content Through Text (PACT) 


PACT is an instructional practice that includes five 

interrelated components that teachers use over the 

course of a 10-day unit: 

• Comprehension Canopy: An overarching 
question is introduced on day one and reviewed 
each day as new knowledge is gained. 

• Essential Words: Four to five key terms are 
introduced and then reviewed throughout the unit. 

• Knowledge Acquisition: Each day, students 
read in various settings (whole class, small group, 
paired, and individually) and record notes from 
their readings and teacher presentations in a log. 

• Team Based Learning (TBL) Comprehension 
Check: Students complete regular multiple choice 
and open-ended knowledge checks. Students 
complete the questions on their own and then 
verify their answers in groups, using resource 
materials to answer questions, when necessary. 

• TBL Knowledge Acquisition: At the end of 

a unit, students engage in debate, requiring 
students to use textual evidence, think critically, 
evaluate team member’s contributions, synthesize 
perspectives, present a final written product to 
entire class, and respond to feedback. 
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What did the study find? 

The study reported, and the WWC confirmed, that 
students in the classrooms where teachers used 
PACT for social studies instruction performed signifi- 
cantly better than students in comparison classrooms 
on the three outcome measures assessed by the 
authors. Students in the PACT classrooms scored 
higher than students in the comparison conditions in 
the reading and social studies comprehension and 
social studies knowledge domains. 


WWC Rating 


The research described in this 
report meets WWC evidence 
standards without reservations 

Strengths: This study is a well-implemented 
randomized controlled trial. 
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Appendix A: Study details 

Vaughn, S., Swanson, E. A., Roberts, G., Wanzek, J., Stillman-Spisak, S. J., Solis, M., & Simmons, D. 
(2013). Improving reading comprehension and social studies knowledge in middle school. Reading 
Research Quarterly, 48(1) 77-93. 


Setting 

The study was conducted in two middle schools in Texas in a near-urban district with a diverse 


student population. 

Study sample 

Study authors conducted random assignment in two stages. First, eighth-grade students were 
randomly assigned to 27 social studies classes taught by five teachers. Then, each teacher’s 
classes were randomly assigned to either the intervention or comparison condition, such that 
each teacher taught intervention and comparison classrooms. A total of 16 classes (contain- 
ing 261 students) were assigned to the intervention condition and 1 1 classes (containing 158 
students) to the comparison condition— all of which were included in the analysis. The analysis 
sample contained 203 to 218 students in the intervention condition and 119 to 126 students 
in the comparison condition, depending on the outcome. Of the students participating, 47.5% 
were male, and 23.2% received free or reduced-price lunch. Approximately 52.7% were Cau- 
casian, 29.8% were Hispanic, 8.8% were African American, 5% were Asian, and 3.6% were 
two or more races. 

Intervention 

group 

Teachers implemented PACT during three distinct 10-day text-based instructional units (Colo- 
nial America, Road to Revolution, and Revolutionary War) over a period of 6-8 weeks. During 
each unit, the teachers used these five components to deliver the content material: Compre- 
hension Canopy, Essential Words, Knowledge Acquisition, Team Based Learning (TBL) Com- 
prehension Check, and TBL Knowledge Acquisition. 

• Comprehension Canopy: An overarching question is introduced on day one and reviewed 
each day as new knowledge is gained. 

• Essential Words: Four to five key terms are introduced and then reviewed throughout the unit. 

• Knowledge Acquisition: Each day, students read in various settings (whole class, small 
group, paired, and individually) and record notes from their readings and teacher presenta- 
tions in a log. 

• Team Based Learning (TBL) Comprehension Check: Students complete regular multiple 
choice and open-ended knowledge checks. Students complete the questions on their own 
and then verify their answers in groups, using resource materials to answer questions, 
when necessary. 

• TBL Knowledge Acquisition: At the end of a unit, students engage in debate, requiring stu- 
dents to use textual evidence, think critically, evaluate team member’s contributions, synthe- 
size perspectives, present a final written product to entire class, and respond to feedback. 

TBL uses heterogeneous groupings of students as a means to engage students in conver- 
sations about the course content and assessments. In TBL, students took on roles to work 
efficiently as a team: team leader (keeps team on task), text source facilitator (guides team 
to find the best text evidence to support an answer), synthesizer (helps team resolve problems 
by synthesizing ideas), and product manager (organizes materials and final products). 
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Comparison 

group 


Outcomes and 
measurement 


Support for 
implementation 


Reason for 
review 


Teachers provided the same social studies content as in the intervention condition but without 
explicit use of the PACT instructional practice components. Some elements of the Compre- 
hension Canopy, Essential Words, and Knowledge Acquisition techniques may have been 
part of the typical teaching style, but neither TBL Comprehension Checks nor TBL Knowledge 
Acquisition components were typically taught in the comparison classrooms. 

Three outcome measures were administered immediately following the intervention: (a) the 
Gates-MacGinitie Reading Comprehension subtest, Fourth Edition, (b) the Assessment of 
Social Studies Knowledge (ASK) Knowledge Acquisition subtest, and (c) the ASK Reading 
Comprehension in Social Studies subtest. For a more detailed description of these outcome 
measures, see Appendix B. 

Prior to the intervention, participating teachers received 10 hours of professional development 
over the course of 2 days. Topics included how to implement the intervention and how to 
prevent cross contamination, since teachers taught both intervention and comparison classes. 
Researchers also visited classes and teacher planning sessions to offer additional support. 
In-class visits occurred at least four times during the first unit, two times during the second 
unit, and once during the last unit. 

This study was identified for review by the WWC because it was supported by a grant to 
The University of Texas at Austin’s Meadows Center for Preventing Educational Risk (Principal 
Investigator: Sharon Vaughn) from the National Center for Education Research (NCER) at the 
Institute of Education Sciences (IES). 
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Appendix B: Outcome measures for each domain 


Reading comprehension 

Gates-MacGinitie Reading 
Comprehension subtest, Fourth Edition 

This Gates-MacGinitie subtest is a 35-minute group-administered assessment of reading comprehension. 

It contains narrative and expository passages of 3-15 sentences. Students read each passage silently, and 
then answer 3-6 multiple-choice questions about the passage. The items increase in difficulty as the student 
progresses through the assessment. Internal consistency for this assessment ranged from 0.91 to 0.93. 

ASK Reading Comprehension in Social 
Studies subtest 

This researcher-developed assessment is an untimed, 21 -item, four-option multiple-choice test consisting of 
three short passages (312-349 words) related to the content covered in the units. Students read each passage 
silently, and then answer seven multiple-choice questions about the passage. It measures students' reading 
comprehension, including their ability to identify main ideas, understand vocabulary in context, identify cause 
and effect, and summarize. Internal consistency for this assessment was 0.85. 

Social studies 

ASK Knowledge Acquisition subtest 

This researcher-developed assessment is an untimed, 46-item, four-option multiple-choice test that measures 
content knowledge on each of the three units. For this study, items were taken from released questions from 
the Texas Assessment of Knowledge and Skills (TAKS) social studies tests, the Massachusetts Comprehensive 
Assessment System (MCAS) social studies tests, and advanced placement tests. Internal consistency for this 
assessment was 0.89. 


Table Notes: The study authors also presented impacts on a researcher-developed follow-up measure administered 4 weeks after the end of the intervention. However, there was 
no information on the reliability of this long-term follow-up assessment, and therefore, it was not included in this WWC report. 
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Appendix C: Study findings for each domain 





Mean 

(standard deviation) 


WWC calculations 


Domain and 
outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Reading comprehension 

Gates-MacGinitie Reading 
Comprehension subtest, 
Fourth Edition 

Grade 8 

322 

students 

0.16 

(1.08) 

-0.07 

(1.09) 

0.23 

0.21 

+8 

0.01 

ASK Reading Comprehension 
in Social Studies subtest 

Grade 8 

339 

students 

0.51 

(1.21) 

0.09 

(1.20) 

0.43 

0.35 

+14 

<0.01 

Domain average for reading comprehension 




0.28 

+11 

Statistically 

significant 

Social studies 

ASK Knowledge Acquisition 
subtest 

Grade 8 

339 

students 

2.28 

(1.90) 

1.71 

(1.73) 

0.57 

0.31 

+12 

0.02 

Domain average for social studies 





0.31 

+12 

Statistically 

significant 


Table Notes: Positive results for mean difference, effect size, and improvement index favor the intervention group; negative results favor the comparison group. The effect size is 
a standardized measure of the effect of an intervention on student outcomes, representing the change (measured in standard deviations) in an average student’s outcome that can 
be expected if the student is given the intervention. The improvement index is an alternate presentation of the effect size, reflecting the change in an average student's percen- 
tile rank that can be expected if the student is given the intervention. The WWC-computed average effect size is a simple average rounded to two decimal places; the average 
improvement index is calculated from the average effect size. 

Study Notes: A correction for multiple comparisons was needed but did not affect whether any of the contrasts were found to be statistically significant. The p-values presented 
here were reported in the original study. The intervention and comparison group means, variances (which were then transformed to standard deviations), and p-values presented 
here were reported in the original study, based on an analysis of a multilevel, multiple-group structural equation model . 2 

The statistical significance of the study's domain average was determined by the WWC; the study is characterized as having a statistically significant positive effect for both the read- 
ing comprehension and social studies domains because univariate statistical tests are reported for each outcome measure, the effect for all measures within the domain are positive 
and statistically significant, and no effects are negative and statistically significant. 
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Endnotes 

1 Single study reviews examine evidence published in a study (supplemented, if necessary, by information obtained directly from the 
authors]) to assess whether the study design meets WWC evidence standards. The review reports the WWC's assessment of whether 
the study meets WWC evidence standards and summarizes the study findings following WWC conventions for reporting evidence on 
effectiveness. This study was reviewed using the single study review protocol, version 2.0. 

2 These WWC-computed effect sizes are based on author-reported analyses of latent variables, rather than observed scores typically 
reported in studies reviewed by the WWC. 

Recommended Citation 

U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse. (2013, August). 

WWC review of the report: Improving reading comprehension and social studies knowledge in middle school. 
Retrieved from http://whatworks.ed.gov 
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Glossary of Terms 

Attrition 


Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Improvement index 


Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Single-case design 
(SCD) 

Standard deviation 


Statistical significance 
Substantively important 


Attrition occurs when an outcome variable is not available for all participants initially assigned 
to the intervention and comparison groups. The WWC considers the total attrition rate and 
the difference in attrition rates across groups within a study. 

If intervention assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review if it falls within the scope of the review protocol and uses either 
an experimental or matched comparison group design. 

A demonstration that the analysis sample groups are similar on observed characteristics 
defined in the review area protocol. 

Along a percentile distribution of students, the improvement index represents the gain 
or loss of the average student due to the intervention. As the average student starts at 
the 50th percentile, the measure ranges from -50 to +50. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust 
the statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which subjects are assigned 
to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which investigators randomly assign 
eligible participants into intervention and comparison groups. 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 

The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample are spread out over a large range of values. 

Statistical significance is the probability that the difference between groups is a result of 
chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelihood that the difference is due to chance is less than 5% (p < 0.05). 

A substantively important finding is one that has an effect size of 0.25 or greater, regardless 
of statistical significance. 


Please see the WWC Procedures and Standards Handbook (version 2.1) for additional details. 


August 2013 


Page 8 


